Automatic clinical workflow that recognizes and analyzes 2d and doppler modality echocardiogram images for automated cardiac measurements and diagnosis of cardiac amyloidosis and hypertrophic cardiomyopathy

ABSTRACT

A computer-implemented method for automated diagnosis of cardiac amyloidosis (CA) and hypertrophic cardiomyopathy (HCM) performed by an automated workflow engine executed by at least one processor includes separating a plurality of echocardiogram (echo) images a heart according to 2D images and Doppler modality images. The 2D images are classified by view type, including A4C video. The 2D images are segmented to produce segmented A4C images having a segmentation mask over the left ventricle. Phase detection is performed on the segmented A4C images to determine systole and diastole endpoints per cardiac cycle. Disease classification is performed on beat-to-beat A4C images for respective cardiac cycles. The cardiac cycle probability scores generated for all of the cardiac cycles are aggregated for each A4C video, and the aggregated probability scores for all the A4C videos are combined to generate a patient-level conclusion for CA and HCM.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. in-part of co-pendingpatent application Ser. No. 17/219,611, filed Mar. 31, 2021, which is acontinuation in-part of U.S. patent application Ser. No. 16/216,929,filed on Dec. 11, 2018, which issued on Apr. 28, 2020 as U.S. Pat. No.10,631,828, the entire disclosures of which are both assigned to theassignee of the present application and incorporated herein byreference.

TECHNICAL FIELD

The disclosed embodiments relate to image and video classification fordisease prediction, and more specifically, to an automatic clinicalworkflow that recognizes and analyzes 2D and Doppler modalityechocardiogram images for automated diagnosis of cardiac amyloidosis andhypertrophic cardiomyopathy.

BACKGROUND

Cardiovascular disease including heart failure is a major health problemaccounting for about 30% of human deaths worldwide. Heart failure isalso the leading cause of hospitalization in adults over the age of 65years. Echocardiography is an important diagnostic aid in cardiology forthe morphological and functional assessment of the heart. In a typicalpatient echocardiogram (echo) examination, a clinician called asonographer places an ultrasound device against the patient's chest tocapture a number of 2D images/videos of the patients' heart. Reflectedsound waves reveal the inner structure of the heart walls and thevelocities of blood flows. The ultrasound device position is variedduring an echo exam to capture different anatomical sections as 2Dslices of the heart from different viewpoints or views. The clinicianhas the option of adding to these 2D images a waveform captured fromvarious possible modalities including continuous wave Doppler, m-mode,pulsed wave Doppler and pulsed wave tissue Doppler. The 2D images/videosand Doppler modality images are typically saved in DICOM (DigitalImaging and Communications in Medicine) formatted files. Although thetype of modality is partially indicated in the metadata of the DICOMfile, the ultrasound device position in both the modality and 2D views,which is the final determinant of which cardiac structure has beenimaged, is not.

After the patient examination, a clinician/technician goes through theDICOM files, manually annotates heart chambers and structures like theleft ventricle (LV) and takes measurements of those structures. Theprocess is reliant on the clinicians' training to recognize the view ineach image and make the appropriate measurements. In a follow upexamination, a cardiologist reviews the DICOM images and measurements,compares them to memorized guideline values and makes a diagnosis basedon the interpretation made from the echocardiogram.

The current workflow process for analyzing DICOM images, measuringcardiac structures in the images and determining, predicting andprognosticating heart disease is highly manual, time-consuming and errorprone. Because the workflow process is so labor intensive, more than 95%of the images available in a typical patient echocardiographic study arenever annotated or quantified. The view angle or Doppler modality typeby which an image was captured is typically not labelled, which meansthe overwhelming majority of stored DICOMs from past patient studies andclinical trials do not possess the basic structure and necessaryidentification of labels to allow for machine learning on this data.

There has been a recent proposal for automated cardiac imageinterpretation to enable low-cost assessment of cardiac function bynon-experts. Although the proposed automated system holds the promise ofimproved performance compared to the manual process, the system hasseveral shortfalls. One shortfall is that the system only recognizes 2Dimages. In addition, although the proposed system may distinguishbetween a normal heart and a diseased heart, the proposed system isincapable of distinguishing hearts having similar-looking diseases.Consequently, the number of heart diseases identified by the proposedsystem is very limited and requires manual intervention to identifyother types of heart diseases.

For example, heart failure has been traditionally viewed as a failure ofcontractile function and left ventricular ejection fraction (LVEF) hasbeen widely used to define systolic function, assess prognosis andselect patients for therapeutic interventions. However, it is recognizedthat heart failure can occur in the presence of normal or near-normalEF: so-called “heart failure with preserved ejection fraction (HFPEF)”which accounts for a substantial proportion of clinical cases of heartfailure. Heart failure with severe dilation and/or markedly reduced EF:so-called “heart failure with reduced ejection fraction (HFREF)” is thebest understood type of heart failure in terms of pathophysiology andtreatment. The symptoms of heart failure may develop suddenly ‘acuteheart failure’ leading to hospital admission, but they can also developgradually.

Timely categorization of heart failure subtype-HFREF or HFPEF andimproved risk stratification are critical for the management andtreatment of heart failure, but the proposed system does not addressthis.

The proposed system is also incapable of generating a prognosis based onthe identified heart disease and would instead require a cardiologist tomanually form the prognosis. The proposed system is further incapable ofstructuring the automated measurements and labelled views acrossmultiple sources of data, to enable training and validation of diseaseprediction algorithms across multiple remote patient cohorts.

Transthoracic echocardiography (TTE) is a common imaging modality toscreen for Cardiac Amyloidosis (CA), and Hypertrophic Cardiomyopathy(HCM). Multi-parametric evaluation of CA and HCM severity is recommendedby current guidelines. There have been other proposals that utilizevideo-based deep learning 3D CNN models for disease detection on A4Cvideos: One proposal implements a single model for detection of CA, anda second proposal implements two separate models for detection of CA andHCM, respectively. In particular, the second proposal performs automaticphase detection followed by beat-to-beat analysis of ventriculardimensions on PLAX videos to detect Left Ventricular Hypertrophy (LVH),then used a 3D CNN to detect CA or HCM if LVH is present. The mainlimitation of these proposals is that they do not include a fullyautomated pipeline comprising of automated view identification andreporting.

Accordingly, there is a need for an improved and fully automaticclinical workflow that recognizes and analyzes both 2D and Dopplermodality echocardiographic images for automated diagnosis of CA and HCM.

BRIEF SUMMARY

The disclosed embodiments provide methods and systems for automateddiagnosis of cardiac amyloidosis (CA) and hypertrophic cardiomyopathy(HCM) performed by an automated workflow engine executed by at least oneprocessor. Aspects of the disclosed embodiments include receiving, froma memory, a plurality of echocardiogram images of a heart. The pluralityof echocardiogram (echo) images according to 2D images and Dopplermodality images. The 2D images are classified by view type, includingA4C video. The 2D images are segmented to produce segmented A4C imageshaving a segmentation mask over the left ventricle. Phase detection isperformed on the segmented A4C images to determine systole and diastoleendpoints per cardiac cycle. Disease classification is performed onbeat-to-beat A4C images for the respective cardiac cycles, and gives aset of probabilities for CA, HCM or neither CA or HCM for each cardiaccycle. The cardiac cycle probability scores generated for all of thecardiac cycles are aggregated for each A4C video, and the aggregatedprobability scores for all the A4C videos are combined to generate apatient-level conclusion for CA and HCM.

According to the method and system disclosed herein, the disclosedembodiments use machine learning to recognize and analyze both 2D andDoppler modality Echocardiographic images for automated measurements andthe diagnosis, prediction and prognosis of heart disease, and the systemcan be deployed in workstation or mobile-based ultrasound point-of-caresystems.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

FIGS. 1A-1C are diagrams illustrating embodiments of a system forimplementing an automated clinical workflow that recognizes and analyzesboth 2D and Doppler modality Echocardiographic images for automatedmeasurements and the diagnosis, prediction and prognosis of heartdisease.

FIG. 2 illustrates architectural layers of the echo workflow engine.

FIG. 3A is a flow diagram illustrating one embodiment of a process forperformed by the echo workflow engine to automatically recognize andanalyze both 2D and Doppler modality echo images for automatedmeasurements and diagnosis of cardiac amyloidosis (CA) and hypertrophiccardiomyopathy (HCM).

FIG. 3B is a diagram illustrating a video-based disease classificationworkflow process for automatically detecting CA and HCM.

FIG. 3C is a diagram illustrating of the video-based diseaseclassification workflow and disease in further detail according to anexemplary embodiment.

FIG. 3D is a diagram illustrating an example report for CA.

FIG. 3E is a diagram illustrating an example report for HCM.

FIG. 4A is a flow diagram illustrating details of the process forautomatically recognizing and analyze both 2D and Doppler modality echoimages to perform automated measurements and the diagnosis, predictionand prognosis of heart disease according to one embodiment.

FIG. 4B is a flow diagram illustrating advanced functions of the echoworkflow engine.

FIG. 5A is a diagram illustrating an example 2D echo image.

FIG. 5B is a diagram illustrating an example Doppler modality image.

FIGS. 6A-6K are diagrams illustrating some example view typesautomatically classified by the echo workflow engine.

FIG. 7 is a diagram illustrating an example 2D image segmented toproduce annotations indicating cardiac chambers, and an example Dopplermodality image segmented to produce a mask and trace waveform.

FIGS. 8A and 8B are diagrams illustrating examples of findingsystolic/diastolic end points.

FIGS. 9A and 9B are diagrams illustrating processing of an imagingwindow in a 2D echo image and the automated detection of out of sectorannotations.

FIGS. 10A and 10B are diagrams graphically illustrating structuralmeasurements automatically generated from annotations of cardiacchambers in a 2D image, and velocity measurements automaticallygenerated from annotations of waveforms in a Doppler modality.

FIG. 11 is a diagram graphically illustrating measurements of globallongitudinal strain that were automatically generated from theannotations of cardiac chambers in 2D images.

FIG. 12A is a diagram graphically illustrating an example set of bestmeasurement data based on largest volume cardiac chambers and the savingof the best measurement data to a repository.

FIG. 12B is a diagram graphically illustrating the input ofautomatically derived measurements from a patient with normal LV EFmeasurements into a set of rules to determine a conclusion that thepatient has normal diastolic function, diastolic dysfunction, orindeterminate.

FIG. 13 is a diagram graphically illustrating the output ofclassification, annotation and measurement data to an example JSON file.

FIG. 14 is a diagram illustrating a portion of an example report showinghighlighting values that are outside the range of internationalguidelines.

FIG. 15 is a diagram illustrating a portion of an example report of MainFindings that may be printed and/or displayed by the user.

FIG. 16A is a diagram showing a graph A plotting PCWP and HFpEF scores.

FIG. 16B is a diagram showing a graph B plotting CV mortality orhospitalization for HF.

DETAILED DESCRIPTION

The disclosed embodiments relate to an automatic clinical workflow thatrecognizes and analyzes 2D and Doppler modality Echocardiographic imagesand videos for automated measurements diagnosis of cardiac amyloidosisand hypertrophic cardiomyopathy. The following description is presentedto enable one of ordinary skill in the art to make and use the inventionand is provided in the context of a patent application and itsrequirements. Various modifications to the disclosed embodiments and thegeneric principles and features described herein will be readilyapparent. The disclosed embodiments are mainly described in terms ofparticular methods and systems provided in particular implementations.However, the methods and systems will operate effectively in otherimplementations. Phrases such as “disclosed embodiment”, “oneembodiment” and “another embodiment” may refer to the same or differentembodiments. The embodiments will be described with respect to systemsand/or devices having certain components. However, the systems and/ordevices may include more or less components than those shown, andvariations in the arrangement and type of the components may be madewithout departing from the scope of the invention. The disclosedembodiments will also be described in the context of particular methodshaving certain steps. However, the method and system operate effectivelyfor other methods having different and/or additional steps and steps indifferent orders that are not inconsistent with the disclosedembodiments. Thus, the present invention is not intended to be limitedto the embodiments shown, but is to be accorded the widest scopeconsistent with the principles and features described herein.

The disclosed embodiments provide method and system for implementing asoftware-based automatic clinical workflow using machine learning thatrecognizes and analyzes both 2D and Doppler modality Echocardiographicimages for automated measurements and diagnosis of cardiac amyloidosis(CA) and hypertrophic cardiomyopathy (HCM), and which can be deployed inworkstation or mobile-based ultrasound point-of-care systems.

The disclosed workflow includes automated identification of 2D videos,such as A4C, for disease classification. The workflow's diseaseclassification model performs beat-to-beat detection of both CA and HCM,allowing a machine learning (ML) model to implicitly study diastolic andsystolic functions in reference with CA and HCM. In addition, amulti-beat algorithm references multiple cardiac cycles wheneverpossible, to generate a more confident final diagnosis of CA/HCM or noCA/HCM.

FIGS. 1A-1C are diagrams illustrating embodiments of a system forimplementing an automated clinical workflow that recognizes and analyzesboth 2D and Doppler modality Echocardiographic images for automatedmeasurements and the diagnosis, prediction and prognosis of heartdisease. FIG. 1A shows a basic standalone configuration for theautomated clinical workflow system 10A and a connected configuration10B. The automated clinical workflow 10A is primarily implemented as asoftware application, referred to as echo workflow engine 12, thatexecutes on a computer 14 operating in a standalone setting,disconnected from other devices on network 26. The computer 14 may beimplemented in any form factor including a workstation, desktop,notebook, laptop server or tablet capable of running an operatingsystem, such as Microsoft Windows® (e.g., Windows 7®, Windows 10®),Apple macOS®, Linux®, Apple iOS®, Android®, and the like.

The computer 14 may include typical hardware components (not shown)including a processor, input devices (e.g., keyboard, pointing device,microphone for voice commands, buttons, touchscreen, etc.), outputdevices (e.g., a display device, speakers, and the like), and wired orwireless network communication interfaces (not shown) for communication.The computer 14 may include internal computer-readable media, such asmemory (not shown) containing computer instructions comprising the echoworkflow engine 12, which implements the functionality disclosed hereinwhen executed by one or more computer processors.

The computer 14 may further include local internal storage for storingone or more databases 16 and an image file archive 18. In oneembodiment, the contents of the image file archive 18 includeechocardiogram image files (also referred to herein as echo images),which in some embodiments may be stored in DICOM (Digital Imaging andCommunications in Medicine) format.

In one embodiment, the computer 14 is in communication with peripheraldevices such a point-of-care (POC) device 25, an ultrasound imagingdevice 24, or both. The POC device 25 is capable of measuring cardiacbiomarkers in POC environments such as an emergency room, intensive careunit, physician's office, an ambulance, a patient setting, and remoteemergency sites.

In one embodiment, the computer 14 is in communication with peripheraldevices such an ultrasound imaging device 24 that capturesechocardiogram images of a patient's organ (e.g., a heart), which maythen be stored as a patient study using the database 16 and image filearchive 18. For example, the computer 14 may be located in a hospital orclinical lab environment where Echocardiography is performed as adiagnostic aid in cardiology for the morphological and functionalassessment of the heart. During a typical patient echocardiogram exam(referred to as a study), a sonographer or technician places theultrasound imaging device 24 against the patient's chest to capture 2Decho images/videos of the heart to help diagnose the particular heartailment. Measurements of the structure and blood flows are typicallymade using 2D slices of the heart and the position of the ultrasoundimaging device 24 is varied during an echo exam to capture differentanatomical sections of the heart from different viewpoints. Thetechnician has the option of adding to these 2D echo images a waveformcaptured from various possible modalities including: continuous waveDoppler, m-mode, pulsed wave Doppler and pulsed wave tissue Doppler. The2D images and Doppler waveform images may be saved as DICOM files.Although the type of modality is sometimes indicated in the metadata ofthe DICOM file, the 2D view is not.

The computer 14 may further include removable storage devices such as anoptical disk 20 and/or a flash memory 22 and the like for storage of theecho images. In some embodiments, the removable storage devices may beused as an import source of echo images and related data structures intothe internal image file archive 18, rather than or in addition to, theultrasound imaging device 24. The removable storage devices may also beused as an archive for echocardiogram data stored in the database 16and/or the image file archive 18.

FIG. 1A also shows an advanced optional embodiment, referred to asconnected configuration 10B, where the computer 14 may be connectedthrough the network 26 and a router 28 to other DICOM based devices,such as DICOM servers 30, network file share devices 32, echoworkstations 34, and/or cloud storage services 36 hosting DICOM files.In the connected configuration 10B, several possible interactions withthe database 16 and the image file archive 18 are possible, as describedbelow.

One possible interaction is to use the cloud storage services 36 as aninternal archive. In case of very large archives consisting of largeamounts of DICOM files, the computer 14 may not have sufficient storageto host all files and the echo workflow engine 12 may be configured touse external network storage of the cloud storage services 36 for filestorage.

Another possible interaction is to use the cloud storage services 36 asan import source by i) selecting a DICOM file set or patient study,which includes the DICOM and Doppler waveforms images and patient dataand examination information. The patient study may also be selected by areserved DICOMDIR file instance, from which the patient, exams and imagefiles are read.

Yet a further possible interaction is to use the DICOM servers 30, thenetwork file share devices 32, echo workstations 34, and/or DICOMclients (of FIG. 1C) acting as DICOM servers (workstations withmodalities CFind, CMove and CStore) in order to retrieve patients, examsand images by performing a CFind operation, followed by a CMoveoperation, to request the remote device to send the images resultingfrom the CFind operation.

Referring now to FIG. 1B, a handheld configuration 10C for the automatedclinical workflow system is shown. In this embodiment, the computer 14of FIG. 1A is implemented as a handheld device 14′, such as a tablet ora mobile phone, connected to a wired or wireless portable ultrasoundscanner probe 24′ that transmits echo images to the handheld device 14′.In such embodiments, the echo workflow engine 12 may be implemented asan application executed by the handheld device 14′.

FIG. 1C illustrates a software as a service (SaaS) configuration 10D forthe automated clinical workflow. In this embodiment, the echo workflowengine 12 is run on a server 40 that is in communication over thenetwork 26 with a plurality of client devices 42. In this embodiment,the server 40 and the echo workflow engine 12 may be part of athird-party service that provides automated measurements and thediagnosis, prediction and prognosis of heart disease to client devices(e.g., hospital, clinic, or doctor computers) over the Internet. Itshould be understood that although the server 40 is shown as a singlecomputer, it should be understood that the functions of server 40 may bedistributed over more than one server. In an alternative embodiment, theserver 40 and the echo workflow engine 12 of FIG. 1C may be implementedas a virtual entity whose functions are distributed over multiple clientdevices 42. Likewise, it should be understood that although the echoworkflow engine 12 is shown as a single component in each embodiment,the functionality of the echo workflow engine 12 may be separated into agreater number of modules/components.

Conventionally, after a patient examination where echo images arecaptured stored, a clinician/technician goes through the DICOM files,manually annotates heart chambers and structures and takes measurements,which are presented in a report. In a follow up examination, a doctorwill review the DICOM images and measurements, compare them to memorizedguideline values and make a diagnosis. Such a process is reliant on theclinicians' training to recognize the view and make the appropriatemeasurements so that a proper diagnosis can be made. Such a process iserror-prone and time consuming.

According to the disclosed embodiments, the echo workflow engine 12mimics the standard clinical practice, processing DICOM files of apatient by using a combination of machine learning, image processing,and DICOM workflow techniques to derive clinical measurements, diagnosespecific diseases, and prognosticate patient outcomes, as describedbelow. While an automated solution to echo image interpretation usingmachine learning has been previously proposed, the solution onlyanalyzes 2D echo images and not Doppler modality waveform images. Thesolution also mentions disease prediction, but only attempts to handletwo diseases (hypertrophic cardiomyopathy and cardiac amyloidosis) andthe control only compares normal patients to diseased patients.

The echo workflow engine 12 of the disclosed embodiments, however,improves on the automated solution by utilizing machine learning toautomatically recognize and analyze not only 2D echo images but alsoDoppler modality waveform images. The echo workflow engine 12 is alsocapable of comparing patients having similar-looking heart diseases(rather than comparing normal patients to diseased patients), andautomatically identifies additional diseases, including both heartfailure with reduced ejection fraction (HFrEF) and heart failure withpreserved ejection fraction (HFpEF). HFrEF is known as heart failure dueto left ventricular systolic dysfunction or systolic heart failure andoccurs when the ejection fraction is less than 40%. HFpEF is a form ofcongestive heart failure where the amount of blood pumped from theheart's left ventricle with each beat (ejection fraction) is greaterthan 50%. Finally, unlike the proposed automated solution, the echoworkflow engine 12 automatically generates a report with the results ofthe analysis for medical decision support, detection of CA and HCM.

Cardiac amyloidosis is a rare, progressive disease characterized by theaccumulation of abnormal proteins called amyloids in the heart tissue.These amyloid deposits can disrupt the normal structure and function ofthe heart, leading to various symptoms and complications. Hypertrophiccardiomyopathy is a genetic heart condition characterized by abnormalthickening (hypertrophy) of the heart muscle, primarily affecting theleft ventricle. This thickening can make it harder for the heart to pumpblood effectively, leading to various symptoms and complications.

Traditional echocardiography to evaluate CA and HCM is highly manual,time consuming, error-prone, limited to specialists, and involves longwaiting times. However, the Artificial Intelligence (AI) approacheddescribed herein allows fully automated, fast and reproducibleechocardiographic image analysis; turning a manual process of 30minutes, 250 clicks, with 21% variability, into an AI-automated processtaking 2 minutes, 1 click, with 0% variability. Such AI-enabledechocardiographic interpretation therefore not only increases efficiencyand accuracy, but also opens the door to decision support fornon-specialists.

FIG. 2 illustrates architectural layers of the echo workflow engine 12.In one embodiment, the echo workflow engine 12 may include a number ofsoftware components such as software servers or services that arepackaged together in one software application. For example, the echoworkflow engine 12 may include a machine learning layer 200, apresentation layer 202, and a database layer 204.

The machine learning layer 200 comprises several neural networks toprocess incoming echo images and corresponding metadata. The neuralnetworks used in the machine learning layer may comprise a mixture ofdifferent classes or model types. In one embodiment, machine learninglayer 200 utilizes a first set of one or more neural networks 200A toclassify 2D images by view type, and uses a second set of neuralnetworks 200B to both extract features from Doppler modality images andto use the extracted features to classify the Doppler modality images byregion (the neural networks used to extract features may be differentthan the neural network used to classify the images). The first set ofneural networks 200A and the second set of neural networks 200B may beimplemented using convolutional neural network (CNN) and may be referredto as classification neural networks or CNNs.

Additionally, a third set of neural networks 200C, including adversarialnetworks, are employed for each classified 2D view type in order tosegment the cardiac chambers in the 2D images and produce segmented 2Dimages. A fourth set of neural networks 200D are used for eachclassified Doppler modality region in order to segment the Dopplermodality images to generate waveform traces. In additional embodiments,the machine learning layer 200 may further include a set of one or moreprediction CNNs for disease prediction and optionally a set of one ormore prognosis CNNs for disease prognosis (not shown). The third andfourth sets of neural networks 200C and 200D may be implemented usingCNNs and may be referred to as segmentation neural networks or CNNs.

In machine learning, a CNN is a class of deep, feed-forward artificialneural network typically used for analyzing visual imagery. Each CNNcomprises an input and an output layer, as well as multiple hiddenlayers. In neural networks, each node or neuron receives an input fromsome number of locations in the previous layer. Each neuron computes anoutput value by applying some function to the input values coming fromthe previous layer. The function that is applied to the input values isspecified by a vector of weights and a bias (typically real numbers).Learning in a neural network progresses by making incrementaladjustments to the biases and weights. The vector of weights and thebias are called a filter and represents some feature of the input (e.g.,a particular shape).

The machine learning layer 200 operates in a training mode to train eachof the CNNs 200A-200D prior to the echo workflow engine 12 being placedin an analysis mode to automatically recognize and analyze echo imagesin patient studies. In one embodiment, the CNNs 200A-200D may be trainedto recognize and segment the various echo image views using thousands ofecho images from an online public or private echocardiogram DICOMdatabase. In one embodiment, the CNNs are trained for both automatedview identification and) automated annotation of relevant CA and HCMviews in the echo images. A CNN may be additionally trained forautomatic phase detection and beat-to-beat direct classification ofdisease from the relevant view. Beat-to-beat direct classification isthe classification of disease when given only the video duration fromone cardiac cycle. The beat-to-beat disease classifications areaggregated to produce a classification for the overall video, and thedisease classifications for all the videos in a patient study arecombined to produce a final disease classification.

The presentation layer 202 is used to format and present information toa user. In one embodiment, the presentation layer is written in HTML 5,Angular 4 and/or JavaScript. The presentation layer 202 may include aWindows Presentation Foundation (WPF) graphical subsystem 202A forimplementing a lightweight browser-based user interface that displaysreports and allows a user (e.g., doctor/technician) to edit the reports.The presentation layer 202 may also include an image viewer 202B (e.g.,a DICOM viewer) for viewing echo images, and a python server 202C forrunning the CNN algorithms and generating a file of the results inJavaScript Object Notation (JSON) format, for example.

The database layer 204 in one embodiment comprises a SQL database 204Aand other external services that the system may use. The SQL database204A stores patient study information for individual patient studiesinput to the system. In some embodiments, the database layer 204 mayalso include the image file archive 18 of FIG. 1A.

FIG. 3A is a flow diagram illustrating one embodiment of a process forperformed by the echo workflow engine 12 to automatically recognize andanalyze both 2D and Doppler modality echo images for automatedmeasurements and diagnosis of cardiac amyloidosis (CA) and hypertrophiccardiomyopathy (HCM). For evaluation of CA and HCM, the 2D image viewtype used is apical 4-chamber (A4C). The process occurs once the echoworkflow engine 12 is trained and placed in analysis mode.

The process may begin by the echo workflow engine 12 receiving from amemory a plurality of echocardiogram images taken by an ultrasounddevice of a heart (block 300). In one embodiment, the patient study mayinclude 70-90 images and videos.

A first module of the echo workflow engine 12 may be used to operate asa filter to separate the plurality of echocardiogram images according to2D images and Doppler modality images based on analyzing image metadata(block 302). The first module analyzes the DICOM tags, or metadata,incorporated in the image, and runs an algorithm based upon the taginformation to distinguish between 2D and modality images, and thenseparate the modality images into either pulse wave, continuous wave,PWTDI or m-mode groupings. A second module of the echo workflow engine12 may perform color flow analysis on extracted pixel data using acombination of analyzing both DICOM tags/metadata and color contentwithin the images, to separate views that contain color from those thatdo not. A third module may anonymize the data by removing metatags thatcontain personal information and cropping the images to exclude anyidentifying information. A fourth module may extract the pixel data fromthe images and converts the pixel data to numpy arrays for furtherprocessing. These functions may be performed by a lesser number or agreater number of models in other embodiments.

Because sonographers do not label the view types in the echo images, oneor more of the neural networks are used to classify the echo images byview type. In one embodiment, the first set of one or more neuralnetworks is used by the echo workflow engine 12 to classify the 2Dimages by view type (block 304); and the second set of neural networksis used by the echo workflow engine 12 to classify the Doppler modalityimages by Doppler modality region (block 306). As shown, the processingof 2D images is separate from the processing of Doppler modality images.In one embodiment, the first and second neural networks may beimplemented using the set of classification convolutional neuralnetworks (CNNs) 200A.

In one specific embodiment, a five class CNN may be used to classify the2D images by view type and an 11 class CNN may be used to classify theDoppler modality images by Doppler modality region. In an alternativeembodiment, the first and second neural networks may be combined intoone neural network. In one embodiment, the classification of the echoimages by view type and regions by the neural networks can be based on amajority voting scheme to determine the optimal answer. For example, avideo can be divided into still image frames, and the workflow enginegenerates for each image frame a classification label, where the labelconstitutes a vote, and the classification label receiving the highestnumber of the votes is applied as the classification of the video.

In one embodiment, the echo workflow engine 12 is trained to classifymany different view types. For example, the echo workflow engine 12 maybe configured to classify any number of different view types including:parasternal long axis (PLAX), apical 2-, 3-, and 4-chamber (A2C, A3C,and A4C), A4C plus pulse wave of the mitral valve, A4C plus pulse wavetissue Doppler on the septal side, A4C plus pulse wave tissue Doppler onthe lateral side, A4C plus pulse wave tissue Doppler on the tricuspidside, A5C plus continuous wave of the aortic valve, A4C+Mmode (TrV),A5C, and by different regions such as: CW AoV (Continuous Wave AorticValve Doppler) and PW LVOT (Pulsed Wave Left Ventricular Outflow TractDoppler).

Based on the classified images, a third set of neural networks is usedby the echo workflow engine 12 to segment regions of interest (e.g.,cardiac chambers) in the 2D images to produce annotated or segmented 2Dimages, including apical 4-chamber (A4C) video images (block 308). Inone embodiment, a neural network trained on videos is used to analyzethe A4C images to detect CA and HCM.

The process of segmentation includes determining locations where each ofthe cardiac chambers begin and end to generate outlines of structures ofthe heart (e.g., cardiac chambers) depicted in each image and/or video.FIG. 6D is a diagram illustrating an example A4C view type, and FIG. 7illustrates a diagram illustrating the A4C view segmented to producesegments or annotations 700.

A fourth set of neural networks is used by the echo workflow engine 12on each classified Doppler modality region to generate waveform tracesfor the Doppler modality images to generate annotated or segmentedDoppler modality images (block 309). The process of segmentationproduces a trace of an outline of the waveform depicting the velocity ofblood flow in a Doppler modality image.

In one embodiment, the third and fourth sets of neural networks may bereferred to as segmentation neural networks and may comprise the set ofsegmentation CNNs 200B and 200C. The choice of segmentation CNN used isdetermined by the view type of the image, which makes the prior correctclassification of view type a crucial step. In a further embodiment,once regions of interest are segmented, a separate neural network can beused to smooth outlines of the segmentations.

In one embodiment, the segmentation CNNs may be trained fromhand-labeled real images or artificial images generated by generaladversarial networks (GANs). One or more segmentation CNNs may betrained for semantic segmentation of the CW AoV and PW LVOT images. Fromthe segmentation of each wave, relevant measurements can be calculated.

Using both the segmented 2D images and the segmented Doppler modalityimages, the echo workflow engine 12 calculates measurements of cardiacfeatures of the heart for predetermined types of diseases (block 310).

The echo workflow engine 12 then generates a conclusion for a diseaseoutcome, including by comparing the calculated measurements of cardiacfeatures to cardiac guidelines for the predetermined types of diseases(block 312).

The echo workflow engine 12 further outputs at least one report to auser showing disease outcomes, including for the predetermined types ofdiseases, ones of the calculated measurements that fall within oroutside of the guidelines (block 314). In one embodiment, two reportsare generated and output: the first report is a list of the calculatedvalues for each measurement with the highest confidence as determined bya rules based engine, highlighting values among the calculatedmeasurements that fall outside of the International guidelines; and thesecond report is a comprehensive list of all measurements calculated onevery image frame of every video, in every view, generating largevolumes of data.

All report data and extracted pixel data may be stored in a structureddatabase to enable machine learning and predictive analytics on imagesthat previously lacked the quantification and labelling necessary forsuch analysis. The structured database may be exported to a cloud-basedserver or may remain on premises (e.g., of the lab owning the images)and can be connected to remotely. By connecting these data sources intoa single network, the disclosed embodiments can progressively traindisease prediction algorithms across multiple network nodes and validatethe algorithms in distinct patient cohorts. In one embodiment, thereports may be electronically displayed to a doctor and/or a patient ona display of an electronic device and/or as a paper report. In someembodiments, the electronic reports may be editable by the user per ruleor role-based permissions, e.g., a cardiologist may be allowed to modifythe report, but a patient may have only view privileges.

FIG. 3B is a diagram illustrating a video-based disease classificationworkflow process for automatically detecting CA and HCA. The process maybegin by segmenting, with a 2D CNN, the left ventricle of the heart inthe image frames of an A4C video to produce segmented A4C images havinga segmentation mask over the left ventricle (block 320—corresponds toFIG. 3A, block 308). A 2D-CNN is a standard convolution neural networkgenerally used on image data.

Automatic phase detection is performed on the segmented A4C images by aphase detection algorithm to determine systole endpoints and diastoleendpoints per cardiac cycle in the A4C video, wherein the images framesof the A4C video between the systole and diastole endpoints comprisebeat-to-beat video images (block 322). In one embodiment, the automaticphase detection may include performing shape analysis on thesegmentation mask in the segmented A4C images to estimate systole anddiastole durations of the video.

FIG. 3C is a diagram illustrating of the video-based diseaseclassification workflow and disease in further detail according to anexemplary embodiment. A4C videos 350 are shown input to 2D CNN 352, theoutput of which is used to determine systole and diastole endpoints 354per cardiac cycle.

Referring again to FIG. 3B, CA/HCM disease classification is performedby a 3D CNN, which receives the beat-to-beat images for respectivecardiac cycles defined by pairs of systole endpoints and diastoleendpoints, and outputs cardiac cycle probability scores per therespective cardiac cycles for at least one of three cardiac cycleoutcomes: i) Cardiac Amyloidosis (CA), ii) Hypertrophic Cardiomyopathy(HCM), and iii) No CA or HCM (block 324). A 3D CNN uses athree-dimensional filter to perform convolutions.

In one embodiment, the 3D CNN model is trained to learn spatiotemporalpatterns in A4C videos to identify CA, HCM or neither. Specifically todo so, when given a cardiac cycle video, the model outputs a vectorcontaining the three probability scores. In one embodiment, the 3D CNNfor CA/HCM disease classification may output probability scores between0-1 for each cardiac cycle outcome. Prior to computing the probabilityscores, the workflow may filter out the A4C video images having lowprobability scores.

Referring again to FIG. 3C, beat-to-beat video images 356 of the A4Cvideo images for a single cardiac cycle are shown input to 3D CNN 358,which outputs a probability or prediction per cardiac cycle for CA, HCMor neither CA or HCM.

Referring again to FIG. 3B, a multi-beat algorithm aggregates thecardiac cycle probability scores generated for all of the cardiac cyclesto generate a A4C video-level conclusion for at least one of threedisease outcomes: i) Cardiac Amyloidosis (CA), ii) HypertrophicCardiomyopathy (HCM), and iii) No CA or HCM (block 326). The workflowmay filter out the A4C video-level disease predictions having lowprobability scores.

Responsive to a patient study including a plurality of A4C videos, themulti-beat algorithm 362 may further combine the A4C video-levelconclusions from the plurality of A4C videos to generate a patient-levelconclusion for the at least one of three disease outcomes (block 328).

Referring again to FIG. 3C, cardiac cycle probability scores 360 areshown input to the multi-beat algorithm 362, which aggregates thecardiac cycle probability scores from all the cycles and all the A4Cvideos and outputs a patient-level CA/HCM conclusion 364. Finally, areport is output to the user showing the patient-level conclusion forthe at least one of three disease outcomes (330).

FIG. 3D is a diagram illustrating an example report for CA, and FIG. 3Eis a diagram illustrating an example report for HCM.

FIG. 4A is a flow diagram illustrating further details of the processfor automatically recognizing and analyze both 2D and Doppler modalityecho images to perform automated measurements and the diagnosis,prediction and prognosis of heart disease according to one embodiment.

The process may begin with receiving one or more patient studies (FIG. 3block 300), which comprises blocks 400-4010. In one embodiment, echoimages from each of the patient studies are automatically downloadedinto the image file archive 18 (block 400). The echo images may bereceived from a directly from local or remote storage source of thecomputer 14. The local storage sources may include internal/externalstorage of the computer 14 including removable storage devices. Theremote storage sources may include the ultrasound imaging device 24, theDICOM servers 30, the network file share devices 32, the echoworkstations 34, and/or the cloud storage services 36 (see FIG. 1A). Inone embodiment, the echo workflow engine 12 includes functions foroperating as a picture archiving and communication server (PACS), whichis capable of handling images from multiple modalities (source machinetypes, one of which is the ultrasound imaging device 24). The echoworkflow engine 12 uses PACS to download and store the echo images intothe image file archive 18 and provides the echo workflow engine 12 withaccess to the echo images during the automated workflow. The format forPACS image storage and transfer is DICOM (Digital Imaging andCommunications in Medicine).

Patient information from each of the patient studies is extracted andstored in the database 16 (block 402). Non-image patient data mayinclude metadata embedded within the DICOM images and/or scanneddocuments, which may be incorporated using consumer industry standardformats such as PDF (Portable Document Format), once encapsulated inDICOM. In one embodiment, received patient studies are placed in aprocessing queue for future processing, and during the processing ofeach patient study, the echo workflow engine 12 queues and checks forunprocessed echo images (block 404). The echo workflow engine 12monitors the status of patient studies and keeps track of them in aqueue to determine which have been processed and which are stillpending. In one embodiment, prioritization of the patient studies in thequeue may be configured by a user. For example, the patient studies maybe prioritized in the queue for processing according to the date of theecho exam, the time of receipt of the patient study or by estimatedseverity of the patient's heart disease.

Any unprocessed echo images are then filtered for having a valid DICOMimage format and non DICOM files in an echo study are discarded (block406). In one embodiment, the echo images are filtered for having aparticular type of format, for example, a valid DICOM file format, andany other file formats may be ignored. Filtering the echo images forhaving a valid image file format enhances the reliability of the echoworkflow engine 12 by rejecting invalid DICOM images for processing.

Any unprocessed valid echo images are then opened and processed in thememory of the computer 14 (block 408). Opening of the echo images forthe patient study in memory of the computer 14 is done to enhanceprocessing speed by echo workflow engine 12. This is in contrast to anapproach of opening the echo files as sub-processes, saving the echofiles to disk, and then reopening each echo image during processing,which could significantly slow processing speed.

The echo workflow engine 12 then extracts and stores the metadata fromthe echo images and then anonymizes the images by blacking out theimages and overwriting the metadata in order to protect patient dataprivacy by covering personal information written on the image (block410). As an example, DICOM formatted image files include metadatareferred to as DICOM tags that may be used to store a wide variety ofinformation such as patient information, Doctor information, ultrasoundmanufacture information, study information, and so on. In oneembodiment, the extracted metadata may be stored in the database 16 andthe metadata in image files is overwritten for privacy.

After receipt and processing of the patient studies, the echo workflowengine 12 separates 2D images from Doppler modality images so the twodifferent image types can be processed by different pipeline flows,described below. In one embodiment, the separating of the images (FIG. 3, block 302) may comprise blocks 412-414. First, the 2D images areseparated from the Doppler modality images by analyzing the metadata(block 412).

FIGS. 5A and 5B are diagrams illustrating an example 2D echo image 500and an example Doppler modality image 502 including a waveform,respectively. The echo workflow engine 12 may determine the image typeby examining metadata/DICOM tags. In one embodiment, information withinthe DICOM tags may be extracted in order to group the images into one ofthe following four classes: 2D only, pulsed-wave (PW), continuous wave(CW), and m-mode. Similarly, the transducer frequency of the ultrasoundimaging device 24 in the metadata may be used to further separate someof the PW images into a fifth class: pulsed-wave tissue doppler imaging(PWTDI).

Referring again to FIG. 4A, the echo workflow engine 12 may also filterout images with a zoomed view, which may also be determined by analyzingthe metadata (block 414). Any of the echo images that has been zoomedduring image capture are not processed through the pipeline because whenzooming, useful information is necessarily left out of the image,meaning the original image would have to be referenced for the missingdata, which is a duplication of effort that slows processing speed.Accordingly, rather than potentially slowing the process in such amanner, the echo workflow engine 12 filters out or discards the zoomedimages to save processing time. In an alternative embodiment, filteringout zoomed images in block 414 may be performed prior to separating theimages in block 412.

After separating the 2D images from the Doppler modality images, theecho workflow engine 12 extracts and converts the image data from eachecho image into numerical arrays (block 416). For the echo images thatare 2D only, the pixel data comprises a series of image frames played insequence to create a video. Because the image frames are unlabeled, theview angle needs to be determined. For the Doppler modality images thatinclude waveform modalities, there are two images in the DICOM file thatmay be used for subsequent view identification, a waveform image and anecho image of the heart. The pixel data is extracted from the DICOM fileand tags in the DICOM file determine the coordinates to crop the images.The cropped pixel data is stored in numerical arrays for furtherprocessing. In one embodiment, blocks 412, 414 and 416 may correspond tothe separating images block 302 of FIG. 3A.

After separating images, the echo workflow engine 12 attempts toclassify each of the echo images by view type. In one embodiment, viewclassification (FIG. 3 blocks 304 and 306) corresponds to blocks418-422.

According to the disclosed embodiments, the echo workflow engine 12attempts to classify each of the echo images by view type by utilizingparallel pipeline flows. The parallel pipeline includes a 2D imagepipeline and a Doppler modality image pipeline. The 2D pipeline flowbegins by classifying, by a first CNN, the 2D images by view type (block418), corresponding to block 304 from FIG. 3A. The Doppler modalityimage pipeline flow begins by classifying, by a second CNN, the Dopplermodality images by view type (block 420), corresponding to block 306from FIG. 3A.

FIGS. 6A-6K are diagrams illustrating some example view typesautomatically classified by the echo workflow engine 12. As statedpreviously, example view types may include parasternal long axis (PLAX),apical 2-, 3-, and 4-chamber (A2C, A3C, and A4C), A4C plus pulse wave ofthe mitral valve, A4C plus pulse wave tissue Doppler on the septal side,A4C plus pulse wave tissue Doppler on the lateral side, A4C plus pulsewave tissue Doppler on the tricuspid side, A5C plus continuous wave ofthe aortic valve, A4C+Mmode (TrV), A5C+PW (LVOT).

Referring again to FIG. 4A, in one embodiment, 2D image classificationis performed as follows. If the DICOM file contains video frames from a2D view, only a small subset of the video frames is analyzed todetermine 2D view classification for more efficient processing. In oneembodiment, the subset of the video frames may range approximately 8-12video frames, but preferably 10 frames are input into one of the CNNs200A trained for 2D to determine the actual view. In an alternativeembodiment, a subset of video frames may be randomly selected from thevideo file. In one embodiment, the CNNs 200A classify each of theanalyzed video frames as one of: A2C, A3C, A4C, ASC, PLAX Modified,PLAX, PSAX AoV level, PSAX Mid-level, Subcostal Ao, Subcostal Hep vein,Subcostal IVC, Subcostal LAX, Subcostal SAX, Suprasternal and Other.

Doppler modality images comprise two images, an echocardiogram image ofthe heart and a corresponding waveform, both of which are extracted fromthe echo file for image processing. In one embodiment, Doppler modalityimage classification of continuous wave (CW), pulsed-wave (PW), andM-mode images is performed as follows. If the DICOM file contains awaveform modality (CW, PW, PWTDI, M-mode), the two extracted images areinput to one of the CNNs 200A trained for CW, PW, PWTDI and M-mode viewclassification to further classify the echo images as one of: CW (AoV),CW (TrV), CW Other, PW (LVOT), PW (MV), PW Other, PWTDI (lateral), PWTDI(septal), PWTDI (tricuspid), M-mode (TrV) and M-mode Other.

There are many more potential classifications available for modalities,but the present embodiments strategically select the classes above,while grouping the remaining potential classes into “Other”, in order tomaximize processing efficiency, while identifying the most clinicallyimportant images for further processing and quantification.Customization of the CNNs 200A occurs in the desired number of layersused and the quantity of filters within each layer. During the trainingphase, the correct size of the CNNs may be determined through repeatedtraining and adjustments until optimal performance levels are reached.

During view classification, the echo workflow engine 12 maintainsclassification confidence scores that indicate a confidence level thatthe view classifications are correct. The echo workflow engine 12filters out the echo images having classification confidence scores thatfail to meet a threshold, i.e., low classification confidence scores(block 422). Multiple algorithms may be employed to deriveclassification confidence scores depending upon the view in question.Anomalies detected in cardiac structure annotations, image quality,cardiac cycles detected, and the presence of image artifacts may allserve to decrease the classification confidence score and discard anecho image out of the automated echo workflow.

With respect to the confidence scores, the echo workflow engine 12generates and analyzes several different types of confidence scores atdifferent stages of processing, including classification, annotation,and measurements (e.g., blocks 422, 434 and 442). For example, poorquality annotations or classifications, which may be due to substandardimage quality, are filtered out by filtering the classificationconfidence scores. In another example, in a patient study the same viewmay be acquired more than once, in which case the best measurements arechosen by filtering out low measurement confidence scores as describedfurther below in block 442. Any data having a confidence score thatmeets a predetermined threshold continues through the workflow. Shouldthere be a duplication of measurements both with high confidence, themost clinically relevant measurement may be chosen.

Next the echo workflow engine 12 performs image segmentation to defineregions of interest (ROI). In computer vision, image segmentation is theprocess of partitioning a digital image into multiple segments (sets ofpixels) to locate and boundaries (lines, curves, and the like) ofobjects. Typically, annotations are a series of boundary linesoverlaying overlaid on the image to highlight segment boundaries/edges.In one embodiment, the segmentation to define ROI (FIG. 3 block 308)corresponds to blocks 426-436.

In one embodiment, the 2D image pipeline annotates, by a third CNN,regions of interests, such as cardiac chambers in the 2D images, toproduce annotated 2D images (block 426). An annotation post process thenerodes the annotations to reduce their dimensions, spline fits outlinesof cardiac structures and adjusts locations of the boundary lines closerto the region of interest (ROIs) (block 427). The 2D image pipelinecontinues with analyzing the ROIs (e.g., cardiac chambers) in theannotated 2D images to estimate volumes and determine key points in thecardiac cycle by finding systolic/diastolic end points (block 430). For2D only views, measurements are taken at the systolic or diastolic phaseof the cardiac cycle, i.e. when the left ventricle reaches the smallestvolume (systole) or the largest volume (diastole). From the 2D videoimages, it must be determined which end points are systolic and whichare diastolic based on the size of the estimated volumes of the leftventricle. For example, a significantly large left ventricle mayindicate a dystonic end point, while a significantly small volume mayindicate a systolic end point. Every video frame is annotated, and thevolume of the left ventricle is calculated throughout the whole cardiaccycle. The frames with minimum and maximum volumes are detected with apeak detection algorithm.

The Doppler modality pipeline analyzes the Doppler modality images andgenerates, by a fourth CNN, a mask and a waveform trace in the Dopplermodality images to produce annotated Doppler modality images (block431).

FIG. 7 is a diagram illustrating an example 2D image segmented in block426 to produce annotations 700 indicating cardiac chambers, and anexample Doppler modality image segmented in block 431 to produce a maskand waveform trace 702.

In one embodiment, the third and fourth CNNs may correspond tosegmentation CNNs 200B. In one embodiment, each of the CNNs 200B used tosegment the 2D images and Doppler modality images may be implemented asU-Net CNN, which is convolutional neural network developed forbiomedical image segmentation. Multiple U-Nets may be used. For example,for 2D images, a first U-Net CNN can be trained to annotate ventriclesand atria of the heart from the A2C, A3C, A4C views. A second U-net CNNcan be trained to annotate the chambers in the PLAX views. For M-modeviews, a third U-Net CNN can be trained to segment the waveform, removesmall pieces of the segments to find likely candidates for the region ofinterest, and then reconnect the segments to provide a full trace of themovement of the mitral valve. For CW views, a fourth U-net CNN can betrained to annotate and trace blood flow. For PW views, a fifth U-netCNN trained to annotate and trace the blood flow. For PWTDI views, asixth U-net CNN can be trained to annotate and trace movement of thetissues structures (lateral/septal/tricuspid valve).

Referring again to FIG. 4A, the Doppler modality pipeline continues byprocessing the annotated Doppler modality images with a sliding windowto identify cycles, peaks are measured in the cycles, and key points inthe cardiac cycle are determined by finding systolic/diastolic endpoints (block 432). Typically, a Doppler modality video may capturethree heart cycles and the sliding window is adjusted in size to blockout two of the cycles so that only one selected cycle is analyzed.Within the selected cycle, the sliding window is used to identifycycles, peaks are measured in the cycles, and key points in the cardiaccycle are determined by finding systolic/diastolic end points.

FIGS. 8A and 8B are diagrams illustrating examples of findingsystolic/diastolic end points in the cardiac cycle, for both 2D andDoppler modalities, respectively, which are key points in order to takeaccurate cardiac measurements.

Referring again to FIG. 4A, in one embodiment, the echo workflow engine12 maintains annotation confidence scores corresponding to the estimatedvolumes, systolic/diastolic end points, identified cycles and measuredpeaks. The echo workflow engine 12 filters out annotated images havingannotation confidence scores that fail to meet a threshold, i.e., lowannotated confidence scores (block 434). Examples of low confidenceannotations may include annotated images having one or more of:excessively small areas/volumes, sudden width changes, out of proportionsectors, partial heart cycles, and insufficient heart cycles.

After images having low annotation confidence scores are filtered out,the echo workflow engine 12 defines an imaging window for each image andfilters out annotations that lie outside of the imaging window (block435).

FIGS. 9A and 9B are diagrams illustrating processing of an imagingwindow in a 2D echo image 900. In FIG. 9A, a shaded ROI 902 is shownhaving unshaded regions or holes 904 therein. Open CV morphologytransformations are used to fill the holes 904 inside the ROI, as shownin In FIG. 9B. Thereafter, a Hough line transformation may be used tofind an imaging window border 906. As is well known, a Hough transformis a feature extraction technique used in digital image processing tofind imperfect instances of objects within a certain class of shapes.After an imaging window border is found, a pixel account of annotationsbeyond the imaging window border is made. Annotations 908 with asignificant number of pixels outside the border of the imaging windoware then discarded.

Referring again to FIG. 4A, in the handheld configuration for theautomated clinical workflow system 10C (See FIG. 1B), the patientstudies may not include Doppler modality images. According to a furtheraspect, the disclosed embodiments accommodate for such handheldconfigurations by using the 2D images to simulate Doppler modalitymeasurements by using Left Ventricular (LV) and Left Atrial (LA) volumemeasurements to derive E, e′ and A (e.g., early and late diastolictransmittal flow and early/mean diastolic tissue velocity) measurements(block 436). In one embodiment, simulating the Doppler modalitymeasurements may be optional and may be invoked based on a softwaresetting indicating the presence of a handheld configuration and/orabsence of Doppler modality images for the current patient study.

Referring again to FIG. 4A, in one embodiment, once cardiac features areannotated during segmentation, the cardiac features are then measuredduring a measurement process (block 310 of FIG. 3A), which in oneembodiment may comprise block 438-448. The process of measuring cardiacfeatures may begin by quantifying a plurality of measurements using theannotations. First, the 2D pipeline measures the 2D images left/rightventricle, left/right atriums, left ventricular outflow (LVOT) andpericardium (block 438). For the Doppler modality images, the Dopplermodality image pipeline measures blood flow velocities (block 440).

More specifically, for A2C, A3C, A4C, and A5C image views, volumetricmeasurements of chamber size are conducted on the systolic and diastolicframes of the video, and image processing techniques mimic a trainedclinician at measuring the volume using the method of disks (MOD). For2D Plax, PSAX (mid-level), PSAX (AoV level), Subcostal, Suprasternal andIVC image views, linear measurements of chamber size and inter-chamberdistances are conducted on the systolic and diastolic frames of thevideo using image processing techniques to mimic the trained clinician.For M-mode image views, from the annotated segments of the movement ofthe Tricuspid Valve, a center line is extracted and smoothed, and thenthe peaks and valleys are measured in order to determine the minimum andmaximum deviations over the cardiac cycle. For PW image views, from theannotations of the blood flow, a mask is created to isolate parts of thewaveform. A sliding window is then run across the trace to identify onefull heart cycle, in combination with heart rate data from the DICOMtags, to use as a template. This template is then used to identify allother heart cycles in the image. Peak detection is then performed oneach cycle and then run through an algorithm to identify which part ofthe heart cycle each peak represents. For CW image views, from theannotations of the trace of the blood flow, curve fitting is performedon the annotation to then quantify the desired measurements. For PWTDIimage views, from the annotations of the movement of the tissue, a maskis created to isolate parts of the waveform. A sliding window is thenrun across the trace to identify one full heart cycle, in combinationwith heart rate data from the DICOM tags, to use as a template. Thistemplate is then used to identify all other heart cycles in the image.Peak detection is then performed on each cycle and then run through analgorithm to identify which part of the heart cycle each peakrepresents.

FIGS. 10A and 10B are diagrams graphically illustrating structuralmeasurements automatically generated from annotations of cardiacchambers in a 2D image, and velocity measurements automaticallygenerated from annotations of waveforms in a Doppler modality,respectively.

The echo workflow engine 12 is configured to produce many measurements.Referring again to FIG. 4A, in one embodiment, the echo workflow engine12 maintains measurement confidence scores corresponding to the measuredleft/right ventricles, left/right atriums, LVOTs, pericardium andmeasured velocities. The echo workflow engine 12 filters out echo imageshaving measurement confidence scores that fail to meet a threshold,i.e., low measurement confidence scores (block 442).

Measurement of cardiac features continues with calculating longitudinalstrain graphs using the annotations generated by the CNNs (block 444).Thereafter, a fifth CNN is optionally used to detect pericardialeffusion 446.

FIG. 11 is a diagram graphically illustrating measurements of globallongitudinal strain that were automatically generated from theannotations of cardiac chambers in 2D images.

Referring again to FIG. 4A, for all remaining non-filtered out data, theecho workflow engine 12 selects as best measurement data themeasurements associated with cardiac chambers with the largest volumes,and saves with the best measurement data, image location,classification, annotation and other measurement data associated withthe best measurements (block 447).

FIG. 12A is a diagram graphically illustrating an example set of bestmeasurement data 1200 based on largest volume cardiac chambers and thesaving of the best measurement data 1200 to a repository, such asdatabase 16 of FIG. 1A.

Referring again to FIG. 4A, the echo workflow engine 12 then generatesconclusions by inputting the best measurement data 1200 to a set ofrules based on international measurement guidelines to generateconclusions for medical decisions support (block 448).

FIG. 12B is a diagram graphically illustrating the input of normal LV EFmeasurements into a set of rules to determine a conclusion that thepatient has normal diastolic function, diastolic dysfunction, orindeterminate.

Referring again to FIG. 4A, after the conclusions are generated, areport is generated and output (FIG. 3 block 314), which may compriseblocks 450-456. Report generation may begin by the echo workflow engine12 outputting the best measurement data 1200 to a JSON file forflexibility of export to other applications (block 450).

FIG. 13 is a diagram graphically illustrating the output ofclassification, annotation and measurement data to an example JSON file.

Referring again to FIG. 4A, a lightweight browser-based user interface(UI) is displayed showing a report that visualizes the best measurementdata 1200 from the JSON file and that is editable by a user (e.g.,doctor/technician) for human verification (block 452). As is well known,a lightweight web browser is a web browser that is optimized to reduceconsumption of system resources, particularly to minimize memoryfootprint, and by sacrificing some of the features of a mainstream webbrowser. In one embodiment, any edits made to the data are stored in thedatabase 16 and displayed in the UI.

In order to make clinically relevant suggestion to the user,measurements associated with the best measurement data 1200 areautomatically compared to current international guideline values and anyout-of-range values are highlighted for the user (block 454).

FIG. 14 is a diagram illustrating a portion of an example report showinghighlighting values that are outside the range of internationalguidelines.

Referring again to FIG. 4A, the user is provided with an option ofgenerating a printable report showing an automated summary of MainFindings (i.e., a conclusion reached after examination) and underliningmeasurements of the patient's health (block 456).

FIG. 15 is a diagram illustrating a portion of an example report of MainFindings that may be printed and/or displayed by the user.

In one embodiment, the automated workflow of the echo workflow engine 12may end at block 456. However, in further aspects of the disclosedembodiments, the process may continue with advance functions, asdescribed below.

FIG. 4B is a flow diagram illustrating advanced functions of the echoworkflow engine. According to this embodiment, the echo workflow engine12, takes as inputs values of specific measurements that wereautomatically derived using machine learning (see block 310), andanalyzes the specific measurements to determine diseasediagnosis/prediction/prognosis versus both disease and matched controlsand normal patient controls (block 460). In one embodiment, the echoworkflow engine 12 may use the disease predictions to perform diagnosisof any combination of: cardiac amyloidosis, hypertrophic cardiomyopathy,restrictive pericarditis, cardiotoxicity, early diastolic dysfunctionand Doppler free diastolic dysfunction assessment (block 462). Aprognosis in the form of an automated score may then be generated topredict mortality and hospitalizations in the future (block 464).

Echocardiography is key for the diagnosis of heart failure withpreserved ejection fraction (HFpEF). However, existing guidelines aremixed in their recommendations for echocardiogram criteria and none ofthe available guidelines have been validated against gold-standardinvasive hemodynamic measurements in HFpEF.

According to one embodiment, the echo workflow engine 12 furthergenerates a diagnostic score for understanding predictions (block 466).Using machine learning, the echo workflow engine 12 validates thediagnostic score against invasively measured pulmonary capillary wedgepressure (PCWP) and determines the prognostic utility of the score in alarge HFpEF cohort.

In one embodiment, the echo workflow engine 12, takes as the inputsvalues of specific measurements that were automatically derived usingmachine learning workflow, and analyzes the input values using an HFpEFalgorithm to compute the HFpEF diagnostic score.

Recognizing that hypertensive heart disease is the most common precursorto HFpEF and has overlapping echocardiogram characteristics with HFpEF,echocardiogram features of 233 patients with HFpEF (LVEF≥50%) wascompared to 273 hypertensive controls with normal ejection fraction butno heart failure. An agnostic model was developed using penalizedlogistic regression model and Classification and Regression Tree (CART)analysis. The association of the derived echocardiogram score withinvasively measured PCWP was investigated in a separate cohort of 96patients. The association of the score with the combined clinicaloutcomes of cardiovascular mortality of HF hospitalization wasinvestigated in 653 patients with HFpEF from the Americas echocardiogramsub study of the TOPCAT trial.

According to one embodiment, left ventricular ejection fraction(LVEF<60%), peak TR velocity (>2.3 m/s), relative wall thickness(RWT>0.39 mm), interventricular septal thickness (>12.2 mm) and E wave(>1 m/s) are selected as the most parsimonious combination of variablesto identify HFpEF from hypertensive controls. A weighted score (range0-9) based on these 5 echocardiogram variables had a combined area underthe curve of 0.9 for identifying HFpEF from hypertensive controls.

FIGS. 16A and 16B are diagrams showing a graph A plotting PCWP and HFpEFscores, and a graph B plotting CV mortality or hospitalization for HF,respectively. Graph A shows that in the independent cohort, the HFpEFscore was significantly associated with PCWP in patients with HFpEF(R²=0.22, P=0.034). Graph B shows that a one-point increase wasassociated with a 12% increase in risk (hazard ratio [HR] 1.12; 95% CI1.02-1.23, P=0.015) for the combined outcome after multivariablecorrection.

According to the disclosed embodiments, the echocardiographic score candistinguish HFpEF from hypertensive controls and is associated withobjective measurements of severity and outcomes in HFpEF.

A method and system for implementing a software-based automatic clinicalworkflow using machine learning that recognizes and analyzes both 2D andDoppler modality echocardiographic images for automated measurements anddiagnosis of cardiac amyloidosis and hypertrophic cardiomyopathy, andwhich can be deployed in workstation or mobile-based ultrasoundpoint-of-care systems has been disclosed. The present disclosure hasbeen described in accordance with the embodiments shown, and there couldbe variations to the embodiments, and any variations would be within thespirit and scope of the present invention. For example, the disclosedembodiments can be implemented using hardware, software, a computerreadable medium containing program instructions, or a combinationthereof. Accordingly, many modifications may be made by one of ordinaryskill in the art without departing from the spirit and scope of theappended claims.

We claim:
 1. A computer-implemented method for automated diagnosis ofcardiac amyloidosis (CA) and hypertrophic cardiomyopathy (HCM) performedby an automated workflow engine executed by at least one processor, themethod comprising: receiving, from a memory, a plurality ofechocardiogram images a heart; separating the plurality ofechocardiogram (echo) images according to 2D images and Doppler modalityimages; classifying the 2D images by view type, including apical4-chamber (A4C) video; segmenting, by a 2D convolutional neural networks(CNN), a left ventricle of the heart in image frames of the A4C video toproduce segmented A4C images having a segmentation mask over the leftventricle; performing automatic phase detection on the segmented A4Cimages to determine systole endpoints and diastole endpoints per cardiaccycle in the A4C video, wherein the images frames of the A4C videobetween the systole and diastole endpoints comprise beat-to-beat videoimages; performing, by a 3D CNN, CA and HCM disease classification byreceiving the beat-to-beat images for respective cardiac cycles, andoutputting cardiac cycle probability scores per the respective cardiaccycles for at least one of three cardiac cycle outcomes: i) CA, ii) HCM,and iii) No CA or HCM; aggregating the cardiac cycle probability scoresgenerated for the respective cardiac cycles to generate a A4Cvideo-level conclusion for at least one of three disease outcomes: i)CA, ii) HCM, and iii) No CA or HCM; responsive to a patient studyincluding a plurality of A4C videos, combining A4C video-levelconclusions from the plurality of A4C videos to generate a patient-levelconclusion for the at least one of three disease outcomes; andoutputting a report showing the patient-level conclusion for the atleast one of three disease outcomes.
 2. The method of claim 1, whereinreceiving, from a memory, the plurality of echo images, furthercomprises: receiving, by the processor, the plurality of echo imagesdirectly from a local or remote source, including an ultrasound device;storing the plurality of echo images in an image archive; and openingthe stored echo images in the memory for processing.
 3. The method ofclaim 1, wherein separating the echo images further comprises: analyzingmetadata incorporated in the echo images to distinguish between the 2Dand the Doppler modality images; separating the Doppler modality imagesinto either pulse wave, continuous wave, PWTDI or m-mode groupings;performing color flow analysis on extracted pixel data using acombination of the metadata and color content within the echo images toseparate views that contain color from those that do not; removing fromthe echo images any metatags that contain personal information andcropping the echo images to exclude any identifying information; andextracting pixel data from the echo images and converting the pixel datato numpy arrays for further processing.
 4. The method of claim 1,wherein classifying the 2D images and the Doppler modality images isbased on a majority voting scheme comprising: dividing a video of a 2Dimage of a Doppler modality image into frames; generating for theframes, classification labels that constitute votes; and applying theclassification label receiving a highest number of the votes as theclassification of the video.
 5. The method of claim 1, furthercomprising: implementing the workflow engine to comprise a first set ofone or more classification CNNs for view classification, a second set ofone or more segmentation CNNs for chamber segmentation and waveformmask/trace, a third set of one or more prediction CNNs for diseaseprediction.
 6. The method of claim 5, wherein the one or moresegmentation CNNs are trained from hand-labeled real images orartificial images generated by general adversarial networks (GANs). 7.The method of claim 1, further comprising: for all non-filtered outdata, selecting as best measurement data the measurements associatedwith cardiac chambers with largest volumes; and saving with the bestmeasurement data, image location, classification, annotation and othermeasurement data associated with the best measurement data.
 8. A system,comprising: a memory storing a plurality of echocardiogram images takenby an ultrasound device of a heart; at least one processor coupled tothe memory; and a workflow engine, which when executed by the at leastone processor is configurable to: receive, from the memory, a pluralityof echocardiogram images of the heart; separate the plurality ofechocardiogram (echo) images according to 2D images and Doppler modalityimages; classify the 2D images by view type, including apical 4-chamber(A4C) video; segment, by a 2D CNN, a left ventricle of the heart inimage frames of the A4C video to produce segmented A4C images having asegmentation mask over the left ventricle; perform automatic phasedetection on the segmented A4C images to determine systole endpoints anddiastole endpoints per cardiac cycle in the A4C video, wherein theimages frames of the A4C video between the systole and diastoleendpoints comprise beat-to-beat video images; perform, by a 3D CNN, CAand HCM disease classification by receiving the beat-to-beat images forrespective cardiac cycles, and outputting cardiac cycle probabilityscores per the respective cardiac cycles for at least one of threecardiac cycle outcomes: i) CA, ii) HCM, and iii) No CA or HCM; aggregatethe cardiac cycle probability scores generated for the respectivecardiac cycles to generate a A4C video-level conclusion for at least oneof three disease outcomes: i) CA, ii) HCM, and iii) No CA or HCM;responsive to a patient study including a plurality of A4C videos,combine the A4C video-level conclusions from the plurality of A4C videosto generate a patient-level conclusion for the at least one of threedisease outcomes; and output a report showing the patient-levelconclusion for at least one of three disease outcomes.
 9. The system ofclaim 8, wherein the workflow engine receives from the plurality of echoimages directly from a local or remote source, including an ultrasounddevice; stores the plurality of echo images in an image archive; andopens the stored echo images in the memory for processing.
 10. Thesystem of claim 8, wherein the workflow engine separates the echo imagesuses metadata incorporated in the echo images to distinguish between the2D and the Doppler modality images; separates the Doppler modalityimages into either pulse wave, continuous wave, PWTDI or m-modegroupings; performs color flow analysis on extracted pixel data using acombination of the metadata and color content within the echo images toseparate views that contain color from those that do not; removes fromthe echo images any metatags that contain personal information andcropping the echo images to exclude any identifying information; andextracts pixel data from the echo images and converts the pixel data tonumpy arrays for further processing.
 11. The system of claim 8, whereinthe workflow engine classifies the 2D images and the Doppler modalityimages based on a majority voting scheme, wherein workflow enginedivides a video of a 2D image of a Doppler modality image into frames;generates for the frames, classification labels that constitute votes;and applies the classification label receiving a highest number of thevotes as the classification of the video.
 12. The system of claim 8,wherein the workflow engine is implemented to comprise a first set ofone or more classification CNNs for view classification, a second set ofone or more segmentation CNNs for chamber segmentation and waveformmask/trace, a third set of one or more prediction CNNs for diseaseprediction.
 13. The system of claim 12, wherein the one or moresegmentation CNNs are trained from hand-labeled real images orartificial images generated by general adversarial networks (GANs). 14.An executable software product stored on a non-transitorycomputer-readable medium containing program instructions forimplementing an automated workflow for diagnosis of cardiac amyloidosis(CA) and hypertrophic cardiomyopathy (HCM), which when executed by a setof one or more processors, are configurable to cause the set of oneprocessors to perform operations comprising: receiving, from a memory, apatient study comprising a plurality of echocardiogram images taken byan ultrasound device of a heart; separating, by a filter, the pluralityof echocardiogram (echo) images according to 2D images and Dopplermodality images based on analyzing image metadata; classifying the 2Dimages by view type, including apical 4-chamber (A4C) video; segmenting,by a 2D CNN, a left ventricle of the heart in image frames of the A4Cvideo to produce segmented A4C images having a segmentation mask overthe left ventricle; performing automatic phase detection on thesegmented A4C images to determine systole endpoints and diastoleendpoints per cardiac cycle in the A4C video, wherein the images framesof the A4C video between the systole and diastole endpoints comprisebeat-to-beat video images; performing, by a 3D CNN, CA and HCM diseaseclassification by receiving the beat-to-beat images for respectivecardiac cycles, and outputting cardiac cycle probability scores per therespective cardiac cycles for at least one of three cardiac cycleoutcomes: i) CA, ii) HCM, and iii) No CA or HCM; aggregating the cardiaccycle probability scores generated for the cardiac cycles to generate aA4C video-level conclusion for at least one of three disease outcomes:i) CA, ii) HCM, and iii) No CA or HCM; responsive to a patient studyincluding a plurality of A4C videos, combining the A4C video-levelconclusions from the plurality of A4C videos to generate a patient-levelconclusion for the at least one of three disease outcomes; andoutputting a report showing the patient-level conclusion for at leastone of three disease outcomes.